Semantic Understanding and Commonsense Reasoning in an Adaptive Photo Agent

In a story telling authoring task, an author often wants to set up meaningful connections between different media, such as between a text and photographs. To facilitate this task, it is helpful to have a software agent dynamically adapt the presentation of a media database to the user's authoring activities, and look for opportunities for annotation and retrieval. Expecting the user to manually annotate photos with keywords greatly burdens the user. Furthermore, even when photos are properly annotated, their retrieval is often very brittle because semantic connections between annotations and the story text that are "obvious" to people (e.g. between “bride” and “wedding”) may easily be missed by the computer. ARIA (Annotation and Retrieval Integration Agent) is a software agent that acts as an assistant to a user writing e-mail or Web pages. As the user types a story, it does continuous retrieval and ranking on a photo database. It can use descriptions in the story text to semi-automatically annotate pictures based on how they are used. The focus of this thesis is threefold: Improving ARIA’s automated annotation capabilities through world-aware semantic understanding of the text; making photo retrieval more robust by using a commonsense knowledge base, Open Mind Commonsense, to make semantic connections between the story text and annotations (e.g. connect "bride" and "wedding"); and learning personal commonsense through the text (e.g. “My sister’s name is Mary.”) that can then be used to improve photo retrieval by enabling personalized semantic connections. Thesis Supervisor: Dr. Henry Lieberman Title: Research Scientist, MIT Media Laboratory